Skip to content

processor_content_modifier: content modifier actions draft.#8656

Closed
pwhelan wants to merge 2 commits intomasterfrom
pwhelan-content-modifier-actions-draft
Closed

processor_content_modifier: content modifier actions draft.#8656
pwhelan wants to merge 2 commits intomasterfrom
pwhelan-content-modifier-actions-draft

Conversation

@pwhelan
Copy link
Contributor

@pwhelan pwhelan commented Apr 1, 2024

Summary

This change adds support for defining multiple actions for a single configured instance of the content_modifier processor.

Here is an example of a content_modifier with multiple actions:

---
pipeline:
  inputs:
    - name: dummy
      dummy: '{"message": "foobar", "foobar": "barfoo"}'
      processors:
        logs:
          - name: content_modifier
            actions:
              - action: insert
                key: foo
                value: bar
              - action: delete
                key: foo
              - action: delete
                key: foobar
              - action: insert
                key: bar
                value: foo
  outputs:
    - name: stdout
      match: "*"

To accomplish this I made several fundamental changes to fluent-bit itself:

  • Add support for lists inside lists for the yaml parser.
  • Add support for using kvlists as values in the yaml parser.
  • Add support for CFL_VARIANT_ARRAY to the config layer.
  • Use a kvlist instead of flb_kv list for processors (could be backported to use flb_kv list).
  • Add a flb_config_map_set_kvlist function which operates on a kvlist and has support for new types
    • FLB_CONFIG_MAP_ARRAY (backed by cfl_array)
    • FLB_CONFIG_MAP_KVLIST (backed by cfl_kvlist)
  • Add a actions with type FLB_CONFIG_MAP_ARRAY to the config_map for processor_content_modifier.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@pwhelan pwhelan marked this pull request as draft April 1, 2024 13:52
@pwhelan
Copy link
Contributor Author

pwhelan commented Apr 2, 2024

I have refactored and reimplemented the yaml side changes in #8661.

tchrono added a commit to tchrono/fluent-bit that referenced this pull request May 26, 2024
@tchrono
config_format: yaml: Support passing arbitrary objects to processors
dc49a51 
This was initially based on
fluent#8656 and
fluent#8661 but ended up making
few different choices:

- Instead of adding FLB_CONFIG_MAP_KVLIST and FLB_CONFIG_MAP_ARRAY as
  config map types, it adds a single FLB_CONFIG_MAP_VARIANT which
  accepts an array or map. The passed array/map can have nested values
  and accept any JSON/YAML types, which are converted to cfl types. The
  processor will simply receive a `cfl_variant` pointer and is
  responsible for releasing the memory later.
- The existing `flb_config_map_set` function is reused by passing the
  `cfl_variant` pointer (from the yaml parsing phase) as the value of a
  `flb_kv`. Then pointer is eventually moved into the processor context.

Signed-off-by: Thiago Padilha <thiago@chronosphere.io>
@tchrono tchrono mentioned this pull request May 26, 2024
Merged
7 tasks
tchrono added a commit to tchrono/fluent-bit that referenced this pull request May 28, 2024
@tchrono
config_format: yaml: Support passing arbitrary objects to processors
b82b55e 
This was initially based on
fluent#8656 and
fluent#8661 but ended up making
few different choices:

- Instead of adding FLB_CONFIG_MAP_KVLIST and FLB_CONFIG_MAP_ARRAY as
  config map types, it adds a single FLB_CONFIG_MAP_VARIANT which
  accepts an array or map. The passed array/map can have nested values
  and accept any JSON/YAML types, which are converted to cfl types. The
  processor will simply receive a `cfl_variant` pointer but doesn't own
  it, the config section is responsible for releasing its memory.
- The existing `flb_config_map_set` function is reused by passing the
  `cfl_variant` pointer (from the yaml parsing phase) as the value of a
  `flb_kv`. Then pointer is eventually passed to the processor context.

Signed-off-by: Thiago Padilha <thiago@chronosphere.io>
tchrono added a commit to tchrono/fluent-bit that referenced this pull request May 29, 2024
@tchrono
config_format: yaml: Support passing arbitrary objects to processors
bfcc74f 
This was initially based on
fluent#8656 and
fluent#8661 but ended up making
few different choices:

- Instead of adding FLB_CONFIG_MAP_KVLIST and FLB_CONFIG_MAP_ARRAY as
  config map types, it adds a single FLB_CONFIG_MAP_VARIANT which
  accepts an array or map. The passed array/map can have nested values
  and accept any JSON/YAML types, which are converted to cfl types. The
  processor will simply receive a `cfl_variant` pointer but doesn't own
  it, the config section is responsible for releasing its memory.
- The existing `flb_config_map_set` function is reused by passing the
  `cfl_variant` pointer (from the yaml parsing phase) as the value of a
  `flb_kv`. Then pointer is eventually passed to the processor context.

Signed-off-by: Thiago Padilha <thiago@chronosphere.io>
tchrono added a commit to tchrono/fluent-bit that referenced this pull request May 30, 2024
@tchrono
config_format: yaml: Support passing arbitrary objects to processors
8c4036b 
This was initially based on
fluent#8656 and
fluent#8661 but ended up making
few different choices:

- Instead of adding FLB_CONFIG_MAP_KVLIST and FLB_CONFIG_MAP_ARRAY as
  config map types, it adds a single FLB_CONFIG_MAP_VARIANT which
  accepts an array or map. The passed array/map can have nested values
  and accept any JSON/YAML types, which are converted to cfl types. The
  processor will simply receive a `cfl_variant` pointer but doesn't own
  it, the config section is responsible for releasing its memory.
- The existing `flb_config_map_set` function is reused by passing the
  `cfl_variant` pointer (from the yaml parsing phase) as the value of a
  `flb_kv`. Then pointer is eventually passed to the processor context.

Signed-off-by: Thiago Padilha <thiago@chronosphere.io>
tchrono added a commit to tchrono/fluent-bit that referenced this pull request Jun 5, 2024
@tchrono
config_format: yaml: Support passing arbitrary objects to processors
7edb1af 
This was initially based on
fluent#8656 and
fluent#8661 but ended up making
few different choices:

- Instead of adding FLB_CONFIG_MAP_KVLIST and FLB_CONFIG_MAP_ARRAY as
  config map types, it adds a single FLB_CONFIG_MAP_VARIANT which
  accepts an array or map. The passed array/map can have nested values
  and accept any JSON/YAML types, which are converted to cfl types. The
  processor will simply receive a `cfl_variant` pointer but doesn't own
  it, the config section is responsible for releasing its memory.
- The existing `flb_config_map_set` function is reused by passing the
  `cfl_variant` pointer (from the yaml parsing phase) as the value of a
  `flb_kv`. Then pointer is eventually passed to the processor context.

Signed-off-by: Thiago Padilha <thiago@chronosphere.io>
edsiper pushed a commit that referenced this pull request Jul 8, 2024
@tchrono @edsiper
config_format: yaml: Support passing arbitrary objects to processors
4b16a1c 
This was initially based on
#8656 and
#8661 but ended up making
few different choices:

- Instead of adding FLB_CONFIG_MAP_KVLIST and FLB_CONFIG_MAP_ARRAY as
  config map types, it adds a single FLB_CONFIG_MAP_VARIANT which
  accepts an array or map. The passed array/map can have nested values
  and accept any JSON/YAML types, which are converted to cfl types. The
  processor will simply receive a `cfl_variant` pointer but doesn't own
  it, the config section is responsible for releasing its memory.
- The existing `flb_config_map_set` function is reused by passing the
  `cfl_variant` pointer (from the yaml parsing phase) as the value of a
  `flb_kv`. Then pointer is eventually passed to the processor context.

Signed-off-by: Thiago Padilha <thiago@chronosphere.io>
@github-actions
Copy link
Contributor

github-actions bot commented Jul 11, 2024

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Jul 11, 2024
@github-actions github-actions bot removed the Stale label Aug 17, 2024
@pwhelan
Copy link
Contributor Author

pwhelan commented Sep 23, 2024

I believe this works has been subsumed by other PRs, closing for now.

@pwhelan pwhelan closed this Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich

@fujimotos fujimotos Awaiting requested review from fujimotos

@koleini koleini Awaiting requested review from koleini

@cosmo0920 cosmo0920 Awaiting requested review from cosmo0920 cosmo0920 will be requested when the pull request is marked ready for review cosmo0920 is a code owner

Assignees

No one assigned

Labels

docs-required

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pwhelan